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A METHOD OF ANNOTATING AN IMAGE 
Technical Field of the Invention 

The present invention relates generally to the field of multi-media and in 
particular, to a method and apparatus for generating metadata based on multi-media 
content. The invention also relates to a computer program product including a computer 
readable medium having recorded thereon a computer program for generating metadata 
based on multi-media content. 

Background Art 

The popularity of digital photography is ever increasing as digital cameras 
become more accessible and the production and editing capabilities of digital images 
improves. Further, the Internet use of digital images is increasing rapidly. Large 
databases of digital images are being assembled for both personal and commercial use. 
As with conventional photography, the need to annotate and catalogue the ever increasing 
number of digital images is of paramount importance in order to allow ease of access and 
use. One method of facilitating the annotation of digital images is to generate "metadata" 
with the image. Metadata is information about the content of digital images or even 
video. For example, an image depicting a beach scene could include a short textual 
description such as "a picture of a beach", the name of a person in the image or a date and 
time that the image was captured. Many Internet image search sites search on metadata 
content descriptions to locate digital images for display. 

Some digital cameras automatically generate a date and time, which is generally 
included in the file name of a digital image when the image is stored and/or displayed 
(e.g. 12Nov_l.jpg). However, the automatically generated date and time says nothing 
about the content and/or event depicted by the digital image and therefore provides only 
limited assistance in annotating, cataloguing and searching for the digital image. 
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Conventionally, a text entry method of generating metadata for digital images 
has been used where a person sorts through a database of digital images, using a 
computer, and stores with each digital image a short textual label indicating a subject 
and/or an event depicted by each digital image. However, the above conventional method 
is very labour intensive and thus time consuming. 

The problem is further exacerbated when a person needs to supply a selection of digital 
images of a particular event and/or subject by e-mail (electronic mail). This involves 
viewing all of the images in a database, selecting the appropriate images and then 
grouping the images together by attaching them one-by-one to an e-mail for sending. 

Disclosure of the Invention 

It is an object of the present invention to substantially overcome, or at least 
ameliorate, one or more disadvantages of existing arrangements. 

According to one aspect of the present invention there is provided a method of 
annotating an image, said method comprising the steps of: 

providing a list of metadata labels and a plurality of icons; 

associating each metadata label with at least one of said icons; 

selecting at least one of said icons depending on at least one subject rendered in 
said image; 

associating said icon with said at least one subject rendered in said image; 

determining the location at which said subject is rendered in said image; and 

storing the metadata label associated with said selected icon and the location of 
said subject as an annotation of said image. 

According to another aspect of the present invention there is provided a method 
of an apparatus for annotating an image, said apparatus comprising: 

metadata providing means for providing a list of metadata labels and a plurality 

of icons; 
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processor means for associating each metadata label with at least one of said 

icons; 

selection means for selecting at least one of said icons depending on at least one 
subject rendered in said image and associating said icon with said at least one subject 
rendered in said image; 

location determination means for determining the location at which said subject 
is rendered in said image; and 

storage means for storing the metadata label associated with said selected icon 
and the location of said subject as an annotation of said image. 

According to still another aspect of the present invention there is provided a 
method of annotating an image, said method comprising the steps of: 

providing a list of metadata labels and a plurality of icons; 

associating each metadata label with at least one of said icons; 

selecting at least one of said icons depending on at least one subject rendered in 
said image; 

associating said icon with said at least one subject rendered in said image, 
wherein a bounding box is constructed about a location at which said subject is rendered 
in said image; 

determining the location at which said subject is rendered in said image based on 
said bounding box; and 

storing the metadata label associated with said selected icon and the location of 
said subject as an annotation of said image. 

Brief Description of the Drawings 

A number of preferred embodiments of the present invention will now be 
described with reference to the drawings, in which: 
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Fig. 1 is a flow diagram of a method of showing a method of annotating an 
image using metadata, in accordance with the preferred embodiment.; 

Fig. 2 is a schematic block diagram of a general purpose computer upon which 
the preferred embodiment of the present invention can be practiced; and 

Fig. 3 shows an example format of an XML file in accordance with the 
preferred embodiment; 

Fig. 4 shows a user interface that can be used to perform the method of Fig. 1 ; 

Fig. 5 shows the user interface of Fig. 4 with bounding boxes overlaying the 
image according to the method of Fig. 1. 

Detailed Description including Best Mode 

Where reference is made in any one or more of the accompanying drawings to 
steps and/or features, which have the same reference numerals, those steps and/or features 
have for the purposes of this description the same function(s) or operation(s), unless the 
contrary intention appears. 

The principles of the preferred method described herein have general 
applicability to generating metadata. However, for ease of explanation, the steps of the 
preferred method are described with reference to digital images. However, it is not 
intended that the present invention be limited to the described method. For example, the 
invention may have application to generating metadata regarding the subject of a textual 
document, a video clip or any computer generated page. 

Fig. 1 is a flowchart showing a method of annotating an image using metadata, 
in accordance with the preferred embodiment of the present invention. The method of 
Fig. 1 is preferably practiced using a conventional general-purpose computer system 200, 
such as that shown in Fig. 2 wherein the processes of Fig. 1 may be implemented as 
software, such as an application program executing within the computer system 200. In 
particular, the steps of the method of annotating an image using metadata, are effected by 
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instructions in the software that are carried out by the computer. The software may be 
divided into two separate parts; one part for carrying out the annotation method; and 
another part to manage the user interface between the method and the user. The software 
may be stored in a computer readable medium, including the storage devices described 
below, for example. The software is loaded into the computer from the computer 
readable medium, and then executed by the computer. A computer readable medium 
having such software or computer program recorded on it is a computer program product. 
The use of the computer program product in the computer preferably effects an 
advantageous apparatus for annotating an image using metadata in accordance with the 
embodiments of the invention. 

The computer system 200 comprises a computer module 201, input devices such 
as a keyboard 202 and mouse 203, output devices including a printer 215 and a display 
device 214. A Modulator-Demodulator (Modem) transceiver device 216 is used by the 
computer module 201 for communicating to and from a communications network 220, for 
example connectable via a telephone line 221 or other functional medium. The 
modem 216 can be used to obtain access to the Internet, and other network systems, such 
as a Local Area Network (LAN) or a Wide Area Network (WAN). 

The computer module 201 typically includes at least one processor unit 205, a 
memory unit 206, for example formed from semiconductor random access memory 
(RAM) and read only memory (ROM), input/output (I/O) interfaces including a video 
interface 207, and an I/O interface 213 for the keyboard 202 and mouse 203 and 
optionally a joystick (not illustrated), and an interface 208 for the modem 216. A storage 
device 209 is provided and typically includes a hard disk drive 210 and a floppy disk 
drive 211. A magnetic tape drive (not illustrated) may also be used. A CD-ROM 
drive 212 is typically provided as a non-volatile source of data. The components 205 
to 213 of the computer module 201, typically communicate via an interconnected bus 204 
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and in a manner which results in a conventional mode of operation of the computer 
system 200 known to those in the relevant art. Examples of computers on which the 
embodiments can be practised include IBM-PC's and compatibles, Sun Sparcstations or 
alike computer systems evolved therefrom. 

Typically, the application program of the preferred embodiment is resident on 
the hard disk drive 210 and read and controlled in its execution by the processor 205. 
Intermediate storage of the program and any data fetched from the network 220 may be 
accomplished using the semiconductor memory 206, possibly in concert with the hard 
disk drive 210. In some instances, the application program may be supplied to the user 
encoded on a CD-ROM or floppy disk and read via the corresponding drive 212 or 21 1, 
or alternatively may be read by the user from the network 220 via the modem device 216. 
Still further, the software can also be loaded into the computer system 200 from other 
computer readable medium including magnetic tape, a ROM or integrated circuit, a 
magneto-optical disk, a radio or infra-red transmission channel between the computer 
module 201 and another device, a computer readable card such as a PCMCIA card, and 
the Internet and Intranets including e-mail transmissions and information recorded on 
websites and the like. An e-mail transmission is an electronic message sent from one 
computer user to another over a digital network such as the Internet or Intranet. The 
message is generally sent using an e-mail application program and is generally in the 
form of a text file containing a textual message. However, an e-mail transmission can 
include other file types such as jpg and .tif files, for example. The foregoing is merely 
exemplary of relevant computer readable mediums. Other computer readable mediums 
may be practiced without departing from the scope and spirit of the invention. 

The method of annotating an image using metadata can alternatively be 
implemented in dedicated hardware such as one or more integrated circuits performing 
the functions or sub functions of Fig. 1 . Such dedicated hardware may include graphic 
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processors, digital signal processors, or one or more microprocessors and associated 
memories. 

The method of annotating an image using metadata, in accordance with the 
preferred embodiment, can now be described with reference to the flowchart of Fig. 1, 
where the method is performed using the computer system 200. The process begins at 
step 101, where a list of metadata labels is provided. The list of metadata labels is 
preferably provided automatically. For example, a list of people's names can be provided 
automatically by extracting the names from an existing database of names, such as an e- 
mail address book. In this instance, names, associated e-mail addresses and other 
personal information can be extracted from the database. Alternatively, a list of metadata 
labels can be manually generated using any known software database application. At the 
next step 103, a plurality of icons are provided. Preferably, the number of icons is equal 
to the number of metadata labels in the list of metadata labels and the icons are preferably 
displayed on the video display 214. The process continues at the next step 105, where 
each label in the list of metadata labels is associated with at least one of the generated 
icons. For example, one of the icons could be associated with the name "Jenny Smith" 
which was included in an imported e-mail address book. The association of each 
metadata label with an icon can be carried out arbitrarily. However, preferably each 
metadata label is associated with a generated icon. 

At the next step 107, a digital image is rendered. The digital image is preferably 
rendered on a display, such as the video display 214. The digital image is preferably 
extracted from a database of images which is stored on the hard disk drive 214 or 
accessed via the computer network 220. The process continues at the next step 109, 
where one of the generated icons is selected according to the content of the image. For 
example, if the image depicts a person called "Liza Hayward", then the icon associated 
with the name Liza Hayward can be selected. The icon is preferably selected by clicking 
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on the appropriate icon using the mouse 203. The icon can preferably be dragged in a 
conventional manner using the mouse 203 and positioned over a desired subject of the 
image. Once the icon is positioned over the desired subject of the image, the user can de- 
select the mouse 203 in the conventional manner. This action is generally referred to as 
"dropping" the icon on the image. At the next step 111, the location at which the subject 
is rendered in the image is determined based on the initial placement of an icon on an 
image. Any known face or object detection method (e.g. colour region growing) can be 
used to detect where the subject is located in the image based on the location of the initial 
placement of the icon. However, in accordance with the preferred embodiment a default 
sized bounding box (e.g. 503 of Fig. 5) is presented to the user over the desired subject 
(e.g. Liza Hayward) such that the bounding box substantially surrounds the subject. The 
bounding box is preferably re-sizeable by the user using the mouse 203 in a conventional 
manner. In this manner, the user can alter the size of the bounding box to totally surround 
the selected subject or to cover only a portion of the subject. Preferably, more than one 
icon can be selected. Alternatively, the bounding box can be automatically resized to fit 
the selected subject. For example, if there is more than one person in the rendered image 
then another icon, which is associated with the name of the other person, can be selected, 
dragged and positioned over the other person, as discussed above. This will result in 
another default sized bounding box (e.g. 505 of Fig. 5) being positioned over the second 
subject, which can preferably be re-sized by the user, as discussed above. In another 
example, a person in the image can be standing with a mountain in the background. 
Therefore, after selecting the icon associated with the person's name, an icon associated 
with the metadata "mountain" can be selected and if desired a bounding box can be 
positioned adjacent to the mountain. 

At the next step 113, the metadata (e.g. the name "Liza Hayward") associated 
with the selected icons is stored as an association list, in a storage device such as the hard 
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disk drive 210, and linked to the rendered image. The position (x,y) and size (width, 
height) of the bounding box (e.g. 503), associated with the subject, are also stored in the 
association list, as at step 113, such that the metadata and bounding box information are 
linked together. Alternatively, two opposite co-ordinate points indicating the size of the 
bounding box can be stored in the association list. Therefore, the bounding box 
information (i.e. position, height and width) indicates the location and size of the subject 
within the rendered digital image. The association list preferably includes a tag indicating 
an association with the rendered image. For example, the list could include the file name 
(e.g. 12Nov_l.jpg) of the corresponding image file for the rendered image. Preferably 
the association list is an Extensible Mark-up Language (XML) file, stored on a hard disk 
drive, such as the hard disk drive 210. XML is a computer language used for associating 
metadata with images. An example of the format of an XML file 301 is shown in Fig. 3. 
As seen in Fig. 3, the XML file includes the file name 300 of an image file, the metadata 
303, and bounding box information 305 associated with the image file. In a further 
embodiment, the metadata and bounding box positional information associated with the 
selected subject can be stored as part of the image file. For example, the .TIF image 
format allows metadata to be stored as part of the image file. 

The process continues at step 115, where a decision is made as to whether any 
more images require annotation. If more images require annotation, then the process 
returns to step 107, where another image is rendered. The process concludes when all of 
the images requiring annotation have been annotated. 

Fig. 4 shows a preferred user interface 401 that can be used to perform the 
method in accordance with the preferred embodiment. The preferred user interface 401 
includes an icon 415 labelled "Input Icon" which can be selected in order to enable the 
user to generate a database of metadata labels, as at step 101 of the preferred method, by 
importing a pre-existing database from the computer network 220 or another application. 
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The preferred user interface 401 also includes an icon 417 labelled "Import Images' 1 
which can be selected in order to enable the user to access the digital images to be 
rendered. The user can preferably import the digital images from a pre-existing database 
which is stored on the hard disk drive 214 or accessed via the computer network 220 or 
another application. The user interface 401 also includes a series of icons 403, which are 
generated as at step 103 of the preferred method. One of the icons 403 can be selected, as 
at step 109 of the preferred method, in order to associate the selected icon with a rendered 
image. The number of generated icons 403 is preferably equal to the number of metadata 
labels in the generated database, which has been previously imported or created by the 
user. The user interface 401 also includes a frame 407 for rendering the images that the 
user wishes to annotate. A forward and reverse button 409 and 411, respectively, are 
provided below the frame 407 to enable the user to successively render all of the images 
included on the database by selecting either forward or reverse with the mouse 203 in a 
conventional manner. 

In accordance with a still further embodiment, the user interface 401 includes 
three icons 419, 421 and 423 labelled "Extract Sub-images", "Display Sub-images" and 
"Rectangles", respectively. The user can select the icon 419 labelled "Extract Sub- 
images" resulting in an image being extracted from the rendered image, based on a 
bounding box which has been dropped on the rendered image using the preferred method 
of Fig. 1. Also, upon selecting the icon 419 labelled "Extract Sub-images", the bounding 
box information (i.e. position, height and width) associated with the bounding box is 
stored in the association list together with the metadata associated with the subject 
overlayed by the bounding box, as at step 113 of the preferred method. After being 
extracted, the image can preferably be stored as an image file (e.g jpg or .tif) in the 
semiconductor memory 206. For example, an image, depicting Liza Hayward, based on 
the bounding box positioned over Liza Hayward in the rendered image, is extracted and 
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stored in the semiconductor memory 206, and the position and size of the bounding box 
are stored in the association list together with the metadata associated with Liza Hayward, 
upon the user selecting the icon 419 labelled "Extract Sub-images". The user can then 
select the icon 421 labelled "Display Sub-images" which results in the extracted image 
being displayed on the user interface 401. As can be seen in Fig. 5, three bounding boxes 
501, 503 and 505 have been dropped on the image rendered in the frame 407 of the user 
interface 401. A user has then selected the icon 419 labelled "Extract Sub-images" 
followed by the icon 421 labelled "Display Sub-images" which has resulted in three 
images 507, 509 and 511 being displayed on the user interface 401 based on the bounding 
boxes 501, 503 and 505. The process of extracting the sub-images in accordance with the 
further embodiment can be carried out sequentially or concurrently after dropping the 
bounding boxes (e.g. 501, 503, 505) onto a rendered image. 

Selecting the icon 423 labelled "Rectangles" preferably results in the bounding 
boxes (e.g. 501, 503 and 505) being removed from the rendered image. However, the 
selection of the icon 423 does not result in the removal of previously stored bounding box 
information and metadata associated with the image. Re-selecting the icon 423 
preferably results in the display of all bounding boxes that have previously been dropped 
onto an image. For example, if a user selects the forward button 409 and the image 
displayed in Fig. 4 is selected from a database of images and rendered in the frame 407, 
then a user can select the icon 423 resulting in all previously stored bounding boxes (e.g. 
501, 503, and 505) being displayed. Re-selecting the icon 423 preferably results in the 
bounding boxes being removed from the image. However, the associated bounding box 
information and metadata previously stored in the association list associated with the 
image will not be affected. 

In accordance with a still further embodiment of the present invention, once 
bounding box positional information and metadata have been stored for an image, 
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pointing to a subject in the image with a mouse pointer (not illustrated) in a conventional 
manner will result in the metadata associated with the subject being displayed. As seen in 
Fig. 5, a user has selected Liza Hayward with the mouse pointer after the bounding box 
503 has been dropped onto the image, resulting in the display of the "tip" 515 including 
Liza Hayward' s name. 

In accordance with a still further embodiment of the present invention, the user 
interface 401 can includes a "content" button (not illustrated) and a "like to send" button 
(not illustrated). The content button can be selected by a user before selecting one of the 
icons 403 in order to indicate that the icon to be subsequently selected will be referring to 
the content of the image. Alternatively, the user can select the like to send button in order 
to indicate that the content of the image will be of interest to a selected person. The icon 
subsequently selected, after selecting the like to send button, will be associated with the 
name of the person to whom the user wishes to send the image. Selecting the content 
button preferably results in a tag (not illustrated) being included in the association list 
stored at step 113, indicating that the metadata associated with the subsequently selected 
icon refers to the content of the image. Similarly, selecting the like to send button 
preferably results in a tag (not illustrated) being included in the association list stored at 
step 113, indicating that the metadata associated with the subsequently selected icon 
refers to a person to whom the user wants to send the image to. 

The icons that are provided at step 103 of the preferred method, are preferably 
default icons having a generic indicator displayed in them. For example, stick figures 
depicting generic people can be displayed in the default icons, as seen in Figs. 4 and 5. 
However, in accordance with the preferred embodiment, if the image rendered at step 107 
is subsequently only associated with one subject at step 109, then the icon used to indicate 
the association can have the generic indicator replaced by a scaled and/or cropped version 
of the rendered image. For example, if the rendered image depicts a person Jenny and the 
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icon associated with the metadata "Jenny" is subsequently selected, then the generic 
indicator of the selected icon can be replaced with a scaled and/or cropped version of the 
image depicting Jenny. Alternatively, the extracted image resulting from a bounding box 
being dropped onto Jenny in an image can be depicted in the icon associated with the 
metadata "Jenny". Replacing the default icons with images improves the user 
friendliness of the icons 403. 

In accordance with a still further embodiment of the present invention, the icons 
provided at step 103 can have e-mail addresses associated with them at step 105. In this 
case, the database of metadata labels, created or imported by the user at step 101, includes 
an e-mail address associated with each name included in the database. When an icon is 
subsequently selected, at step 109 of the preferred method, the metadata associated with 
the selected icon, describing the person's name and e-mail address, is stored in the 
association list at step 113 of the preferred method. Subsequently, if a user wishes to e- 
mail a number of image files to another person, the e-mail application can search through 
the association lists associated with each image file annotated by the user and then send 
the appropriate images to the person. For example, the e-mail application can search 
through the association lists and send all of the images that have the metadata "Jenny" 
associated with them using Jenny's e-mail address. In a further example, if several of the 
lists have been tagged as "like to send", as described above, then the e-mail application 
can select those images and e-mail them to the address of the person indicated by the 
metadata included in the association lists associated with those images. 

Once an image or text document has been annotated using the preferred method, 
a user can perform any number of functions with the annotated material. For example, a 
user can collate a number of annotated images for printing or faxing. Further, a number 
of annotated images can be displayed in sequence based on the content of the images. 
Still further, the annotation can facilitate in later searching for and recovering the image. 
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Still further, the annotation can facilitate in the automatic presentation of images allowing 
a user to zoom in on a subject of interest. Still further, the annotation can facilitate in the 
generation of "tool tips" such that when a user does a conventional mouse roll-over of an 
annotated image, an unknown person in the image can be identified by pointing the 
mouse pointer at that person. Still further, the extraction and display of images in 
accordance with the preferred embodiment can be used to subdivide and/or mosaic an 
image. 

The aforementioned preferred method comprises a particular control flow. There 
are many other variants of the preferred method which use different control flows without 
departing the spirit or scope of the invention. Furthermore one or more of the steps of the 
preferred method can be performed in parallel rather sequentially. 

The foregoing describes only some embodiments of the present invention, and 
modifications and/or changes can be made thereto without departing from the scope and 
spirit of the invention, the embodiments being illustrative and not restrictive. In 
accordance with a still further embodiment of the present invention, based on the initial 
placement of an icon on an image, the embodiments can use any known face or object 
detection method to detect where the subject associated with the icon is in the image. 
Processing can then be carried out, using any known segmentation method, to extract an 
exact spline outline of the subject resulting in positional information being stored for the 
image, as discussed above. 

In accordance with the still further embodiment, the bounding box, discussed 
above, can comprise any number of edges. 

In accordance with a still further embodiment of the present invention, once the 
position of a subject has been determined using use any known face or object detection 
method, a bounding box (not illustrated) can be automatically positioned over the subject 
resulting in positional information being stored, as discussed above. 



495386 CFPI696AUMmcdia57 



l:\ELEC\CISRA\MMEDIA\MMedia57\495386.doc 



- 15 - 

In the context of this specification, the word "comprising" means "including 
principally but not necessarily solely" or "having" or "including" and not "consisting only 
of. Variations of the word comprising, such as "comprise" and "comprises" have 
corresponding meanings. 
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Glaims: The Claims defining the invention are as follows: 

1 . A method of annotating an image, said method comprising the steps of: 

providing a list of metadata labels and a plurality of icons; 
associating each metadata label with at least one of said icons; 
selecting at least one of said icons depending on at least one subject rendered in 
said image; 

associating said icon with said at least one subject rendered in said image; 
determining the location at which said subject is rendered in said image; and 
storing the metadata label associated with said selected icon and the location of 
said subject as an annotation of said image. 

2. The method according to claim 1, further comprising the step of constructing a 
bounding box about locations at which said subject is rendered in said image. 

3. The method according to claim 2, wherein said location of said subject is 
dependent on said bounding box. 

4. The method according to claim 2, further comprising the step of storing the 
location of said bounding box. 

5. The method according to claim 4, wherein the location of said bounding box is 
indicated by at least two co-ordinate points of said bounding box. 

6. The method according to any one of claims 1 to 5, further comprising the step of 
extracting a portion of said image. 
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7. The method according to claim 6 when dependent on claim 2, wherein said 
extracted portion is dependent on said bounding box. 

8. The method according to claim 6, further comprising the step of rendering said 
extracted portion. 

9. The method according to any one of claims 2 to 8, wherein a size of said 
bounding box is user selectable. 

9A. The method according to any one of claims 2 to 8, wherein a size of said 
bounding box is determined automatically. 

10. The method according to claim 2, wherein said bounding box is overlayed on 
said image. 

11. The method according to any one of claims 1 to 10, wherein said metadata is 
displayed upon selecting said subject in said image. 

1 1 A. The method according to claim 1, wherein the location at which said subject is 
rendered in said image is determined based on object detection. 

IIB. The method according to claim 1, wherein the location at which said subject is 
rendered in said image is determined based on object segmentation. 

IIC. The method according to claim 6, wherein said portion of said image is 
segmented in order to determine the location of said subject. 
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12. The method according to any one of claims 1 to 11C, wherein said plurality of 
metadata labels are generated from a pre-existing database of labels. 

13. The method according to any one of claims 1 or 12, wherein the number of said 
icons is equal to the number of said metadata labels. 

14. The method according to any one of claims 1 to 13, wherein said image is 
extracted from a database of images. 

15. The method according to any one of claims 1 to 14, comprising the further step 
of storing a tag indicating an association with said image. 

16. The method according to claim 15, wherein said tag is the file name of said 
image. 

17. The method according to any one of claims 1 to 15, wherein said metadata label 
associated with said selected icon and, said tag are stored in an XML file. 

18. The method according to any one of claim 1 to 17, comprising the further step of 
e-mailing at least said image to at least one other party based on said metadata label 
associated with said selected icon. 

19. The method according to any one of claims 1 to 18, wherein said selected icon is 
associated with a person's name. 
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20. An apparatus for annotating an image, said apparatus comprising: 

metadata providing means for providing a list of metadata labels and a plurality 

of icons; 

processor means for associating each metadata label with at least one of said 

icons; 

selection means for selecting at least one of said icons depending on at least one 
subject rendered in said image and associating said icon with said at least one subject 
rendered in said image; 

location determination means for determining the location at which said subject 
is rendered in said image; and 

storage means for storing the metadata label associated with said selected icon 
and the location of said subject as an annotation of said image. 

21. The apparatus according to claim 20, further comprising bounding box creation 
means for constructing a bounding box about locations at which said subject is rendered 
in said image. 

22. The apparatus according to claim 21, wherein said location of said subject is 
dependent on said bounding box. 

23. The apparatus according to claim 21, wherein said storage means stores the 
location of said bounding box. 

24. The apparatus according to claim 23, wherein the location of said bounding box 
is indicated by at least two co-ordinate points of said bounding box. 
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25. The apparatus according to any one of claims 20 to 24, further comprising means 
for extracting a portion of said image. 

26. The apparatus according to claim 25 when dependent on claim 21, wherein said 
extracted portion is dependent on said bounding box. 

27. The apparatus according to claim 25, further comprising the step of rendering 
said extracted portion. 

28. The apparatus according to any one of claims 21 to 27, wherein a size of said 
bounding box is user selectable. 

30. The apparatus according to claim 21, wherein said bounding box is overlayed on 
said image. 

31 . The apparatus according to any one of claims 21 to 30, wherein said metadata is 
displayed upon selecting said subject in said image. 

31 A. The apparatus according to claim 20, wherein the location at which said subject 
is rendered in said image is determined based on object detection. 

32. The apparatus according to any one of claims 20 to 31 A, wherein said plurality 
of metadata labels are generated from a pre-existing database of labels. 

33. The apparatus according to any one of claims 20 or 32, wherein the number of 
said icons is equal to the number of said metadata labels. 
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34. The apparatus according to any one of claims 20 to 33, wherein said image is 
extracted from a database of images. 

35. The apparatus according to any one of claims 20 to 34, further comprising means 
for storing a tag indicating an association with said image. 

36. The apparatus according to claim 35, wherein said tag is the file name of said 
image. 

37. The apparatus according to any one of claims 20 to 35, wherein said metadata 
label associated with said selected icon and, said tag are stored in an XML file. 

38. The apparatus according to any one of claim 20 to 37, further comprising means 
for e-mailing at least said image to at least one other party based on said metadata label 
associated with said selected icon. 

39. The apparatus according to any one of claims 20 to 38, wherein said selected 
icon is associated with a person's name. 

40. A method of annotating an image, said method comprising the steps of: 
providing a list of metadata labels and a plurality of icons; 
associating each metadata label with at least one of said icons; 

selecting at least one of said icons depending on at least one subject rendered in 
said image; 
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associating said icon with said at least one subject rendered in said image, 
wherein a bounding box is constructed about a location at which said subject is rendered 
in said image; 

determining the location at which said subject is rendered in said image based on 
said bounding box; and 

storing the metadata label associated with said selected icon and the location of 
said subject as an annotation of said image. 



41. A computer readable medium for storing a program for an apparatus which 
processes data, said processing comprising a method of annotating an image, said 
program comprising code for performing the method according to any one of claims 1 to 
19 and 40. 

42. A method of annotating an image, substantially as herein described with 
reference to the accompanying drawings. 



DATED this 28th Day of April 2000 
Canon Kabushiki Kaisha 

Patent Attorneys for the Applicant 
SPRUSON & FERGUSON 



495380 CTPI69GAU Mmcdia57 



l:\CLEOCISRA\MMEDIA\MMcdia57V495386.doc 




1/5 



START 




I 



Generate a list of 
metadata labels. 



I 



Generate a plurality of 
icons. 



I 



Associate each 
metadata with at least 
one of the icons 



3 



Render an image. 



T 



Select at least one of 
the icons and drop the 
icon on the image. 



I 



Determine the location 
of a desired subject 
within the image. 



Store the location and 
metadata associated 
with the desired subject. 



Yes 



101 



103 



105 



107 



109 



111 



113 




END 




FIG. 1 



495386 CFP1696AU Mmedia57 



[i:elec\cisra\Mmedia\Mmedia57] 495386.fm 



2/5 




FIG. 2 



203 



495386 CFP1696AU Mmedia57 



(i:elec\cisra\Mm dia\Mmedia57] 495386 fm 



* 



3/5 



300 



<?xml version = 11 1.0" ?> 

<Entity 1 system 12Nov_l.jpg ND ATA jpeg> 

<Entity 2 system 12Nov_2.jpg NDATA jpeg> 



<image id= "1" subject^ "Liza Hayward" 
position = "(2, 5) (7, 15)" height = "10" width = "5' 
></image> 

<image id= "2" subject= "Jenny^ill"></image> 
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